Papers with IT automation tasks
Large Language Models for IT Automation Tasks: Are We There Yet? (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks rely on synthetic tasks that fail to capture the needs of practitioners who use IT automation tools. |
| Approach: | They evaluate 14 open-source and 3 proprietary LLMs and find that GPT-4.1-Mini achieves the best pass@10 rate of 23.9%, while Claude-3.5-Sonnet achieves best pass @1 performance. |
| Outcome: | The evaluated LLMs perform poorly in 126 tasks and show that they lack state reconciliation capabilities and lack module knowledge. |